Medical EducationVolume 35, Issue 10 p. 920-921 Free Access Control and validity in medical educational research First published: 07 July 2008 https://doi.org/10.1111/j.1365-2923.2001.01033.xCitations: 8 B Jolly Department of Medical Education, University of Sheffield, Division of Clinical Sciences, Northern General Hospital, Herries Road, Sheffield S5 7AU, UK Brian Jolly Sheffield, UK AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Educational research has recently been described as unhelpful in answering real life questions.1 Medical research too has increasingly come under pressure to be better designed, with the randomised controlled trial frequently being hailed as the pre-eminent research design for answering many clinical problems.2 Research in medical education has also come under scrutiny with exhortations to try to achieve quality approaching that of the clinical or laboratory scientist.3,4 In this issue we can see an example of a study aspiring to these ideals.5 Essentially two adequately large groups, constructed using randomised stratification to ensure they were representative, were taught arterial blood gas interpretation using computer or textbook based strategies, within what is commonly called a pre-post test control group design.6 The study had a follow-up three weeks later to look at longer term effects (knowledge retention) – a laudable, but not particularly common, strategy in educational evaluations. Most aspects of control in experimental studies are aimed at achieving what is called internal validity. This is the capacity to be able to state categorically that the effects demonstrated within the study are attributable to the manipulations made by the researcher and not to external or serendipitous factors. Randomisation, the use of control groups, appropriate statistical comparison, and pre- as well as post-testing are all facets of design (among others) that can contribute to this capacity. However, over 30 years ago, Levy7 pointed out the disadvantages of not thinking through the impact that experimental stringency might have on the generalisability of the results of educational research – the so-called external validity of an educational study. Even commonly regarded essential aspects of the design of controlled studies have this impact and should be used with care. For example, the pre-test, when randomisation has taken place, is not absolutely essential. This is because, when used, it technically limits the generalisability of findings only to pre-tested groups – a feature not always appreciated by educationalists and clinicians alike.8,9 This issue may not be important in measurements of accumulated knowledge or skills, but can have a significant impact in attitudinal or health behaviour research. As further examples of such constraints, Levy cited numerous studies from the early 1960s comparing the in vogue methods of film, television and programmed learning against standard lectures. One example involved a study where film alone was being compared to film with interspersed adjunct questions. Because the film was shorter than the combination, to equate the study time of the two groups the researchers showed the film twice to the control group. Levy asks ‘by what educational argument is repetition of the film a plausible and sensible use of the four and a half minutes’ discrepancy between the two methods?’.7 Using such examples he was able to show the fragility of being able to make extrapolations from many experimental studies to the real world. In the Vichitvejpaisal et al. study it is not the pre-test that is the main limit to external validity, but the issue of the context imposed by the controls in an effort to eradicate confounding variables. Mostly, classic design principles were sensibly employed.6 However, the experimenters effectively locked the two groups into their study areas from 7·30 am till 5·30 pm, albeit with access to drinks and food. Quite apart from the ethical robustness of this manoeuvre, which must be doubtful, we need to ask at least two questions. First, ‘how often, in reality, do students utilise learning resources of any type continuously over such a period without recourse to questions and answers, peer group discussion, alternative material, and other adjuncts?’. Second, as a consequence, ‘how much fidelity would the experimental or control conditions have to the ‘real’ use of these modes of delivery in everyday medical education, for the majority of students?’. One feature of the study not discussed is how familiar was the population of students sampled with being asked to study for such extended periods of time in such conditions. If these methods were representative of teaching and learning patterns, then a superbly designed study has been carried out, but one that will probably generalise only to the host institution. The important principle here is that, in educational research, every aspect of control that is introduced may improve internal validity, but also has an impact on external validity. Perhaps a more important question in this case might have involved how students adapted their learning strategies to cope with a 10 hour day anchored to one location with only a single educational framework at their disposal. References 1 Tooley J. Educational Research. A Critique. In: HMI OFSTED Report 1998. London: HMI OFSTED; 1998. Google Scholar 2 Sackett D & Cook RJ. Understanding clinical trials. BMJ 1994; 309: 755 – 6.CrossrefCASPubMedWeb of Science®Google Scholar 3 Hutchinson L. Evaluating and researching the effectiveness of educational interventions. BMJ 1999; 318: 1267 – 9.CrossrefCASPubMedWeb of Science®Google Scholar 4 Wilkes M & Bligh J. Evaluating educational interventions. BMJ 1999; 318: 1269 – 72.CrossrefCASPubMedWeb of Science®Google Scholar 5 Vichitvejpaisal P, Sitthikongsak S, Preechakoon B, Kraiprasit K, Parakkamodom S, Manon C, Petcharatana S. Does computer-assisted instruction really help to improve learning process. Med Educ, 2001; 35: 983 – 989. Wiley Online LibraryCASPubMedWeb of Science®Google Scholar 6 Campbell DT & Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally; 1963. Google Scholar 7 Levy P. New research for new curricula. J. Curriculum Studies 1969; 1: 101 – 8. CrossrefWeb of Science®Google Scholar 8 Sanci LA, Coffey CMM, Veit FCM, Carr-Gregg M, Patton GC, Day N, Bowes G. Evaluation of the effectiveness of an educational intervention for general practitioners in adolescent health care: randomised controlled trial. BMJ 2000; 320: 224 – 9.DOI: 10.1136/bmj.320.7229.224CrossrefCASPubMedWeb of Science®Google Scholar 9 Ker J. Commentary: applying the BMJ’s guidelines on educational interventions. BMJ 2000; 320: 230 – 230.PubMedWeb of Science®Google Scholar Citing Literature Volume35, Issue10October 2001Pages 920-921 ReferencesRelatedInformation
Search from 250 M+ research papersSearch from 250 M+ research papers
Open Access
Jan 1, 2024
Willem Raat + 5